-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Init plugin #3
Conversation
WalkthroughThe repository underwent significant updates to introduce Google Pub/Sub as a jobs driver, enhancing the jobs plugin with cloud-native services support for both AWS and GCP. This includes new configurations, workflows for code analysis and linting, and extensive changes to the pubsubjobs package to manage topics, subscriptions, and message processing efficiently. Additionally, the project saw updates to testing infrastructure, including new PHP test files and mock implementations for robust validation. Changes
Assessment against linked issues
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
Hey @cv65kr 👋 |
Hey @cv65kr 👋 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Review Status
Actionable comments generated: 21
Configuration used: CodeRabbit UI
Files ignored due to path filters (8)
go.mod
is excluded by:!**/*.mod
go.sum
is excluded by:!**/*.sum
go.work
is excluded by:!**/*.work
go.work.sum
is excluded by:!**/*.sum
tests/env/docker-compose-emulator-local.yaml
is excluded by:!**/*.yaml
tests/go.mod
is excluded by:!**/*.mod
tests/go.sum
is excluded by:!**/*.sum
tests/php_test_files/composer.json
is excluded by:!**/*.json
Files selected for processing (22)
- .github/dependabot.yml (1 hunks)
- .github/pull_request_template.md (1 hunks)
- .github/workflows/codeql-analysis.yml (1 hunks)
- .github/workflows/linters.yml (1 hunks)
- .github/workflows/linux.yml (1 hunks)
- .gitignore (1 hunks)
- plugin.go (3 hunks)
- pubsubjobs/config.go (2 hunks)
- pubsubjobs/driver.go (4 hunks)
- pubsubjobs/item.go (7 hunks)
- pubsubjobs/listener.go (1 hunks)
- tests/configs/.rr-declare.yaml (1 hunks)
- tests/configs/.rr-init.yaml (1 hunks)
- tests/configs/.rr-jobs-err.yaml (1 hunks)
- tests/configs/.rr-pq.yaml (1 hunks)
- tests/helpers/helpers.go (1 hunks)
- tests/jobs_test.go (1 hunks)
- tests/mock/logger.go (1 hunks)
- tests/mock/observer.go (1 hunks)
- tests/php_test_files/jobs/jobs_err.php (1 hunks)
- tests/php_test_files/jobs/jobs_ok.php (1 hunks)
- tests/php_test_files/jobs/jobs_ok_pq.php (1 hunks)
Files skipped from review due to trivial changes (1)
- .github/pull_request_template.md
Additional comments: 37
.gitignore (2)
- 15-16: The inclusion of
**/vendor/
andvendor/
in the.gitignore
file is a good practice to avoid committing third-party code to the repository. This change helps keep the repository clean and focused on source code.- 18-18: Adding
.DS_Store
to.gitignore
is a good practice to prevent system-specific files from being committed to the repository, which can clutter the project and is irrelevant to other developers.tests/php_test_files/jobs/jobs_ok.php (1)
- 11-11: Creating a
Consumer
instance without specifying any configuration or connection details directly in the code. Ensure that theConsumer
is correctly configured elsewhere, typically in a configuration file or environment variables, to connect to the appropriate Google Pub/Sub topic.tests/configs/.rr-declare.yaml (1)
- 11-15: The
google-pub-sub
configuration specifies aproject_id
,topic
,skip_topic_declaration
, andhost
. Ensure that thehost
value127.0.0.1:8085
is appropriate for your testing environment, as it seems to point to a locally hosted instance of Google Pub/Sub (possibly emulated). For production environments, this will need to be updated to point to the actual Google Cloud Pub/Sub service.tests/php_test_files/jobs/jobs_ok_pq.php (1)
- 15-15: The
sleep(15);
call simulates a delay in task processing. While this is useful for testing purposes to mimic real-world processing times, ensure that this delay is appropriate for the tests you intend to run. For performance testing or integration tests, the delay might need adjustment.Consider making the sleep duration configurable or documented to ensure test flexibility and clarity.
tests/configs/.rr-init.yaml (2)
- 11-15: The Google Pub/Sub configuration section is well-defined, ensuring that the plugin is correctly configured for testing. However, it's important to verify that the
host
value (127.0.0.1:8085
) matches the expected test environment setup, especially if an emulator is being used for local testing.Verification successful
The Google Pub/Sub emulator is indeed configured to run on
127.0.0.1:8085
, as evidenced by thetests/helpers/helpers.go
andtests/env/docker-compose-emulator-local.yaml
files. This confirms that thehost
value in the.rr-init.yaml
file matches the expected test environment setup.* 31-44: The pipeline configuration for Google Pub/Sub is correctly set up with two test pipelines (`test-1` and `test-2`). Both are configured to use the `google-pub-sub` driver, which aligns with the PR's objectives. However, it's worth noting that `test-2` does not specify a `prefetch` value, unlike `test-1`. Ensure this omission is intentional and aligns with the testing requirements.Scripts Executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify if the emulator for Google Pub/Sub is set up to run on 127.0.0.1:8085 docker ps | grep "google-pub-sub-emulator" | grep "127.0.0.1:8085"Length of output: 138
Script:
#!/bin/bash # Search for mentions of "google-pub-sub-emulator" and "8085" within the repository rg "google-pub-sub-emulator" || echo "No mentions of google-pub-sub-emulator found." rg "8085" || echo "No mentions of port 8085 found."Length of output: 437
.github/dependabot.yml (1)
- 1-33: The configuration for Dependabot is correctly set up to update dependencies for Go modules, Docker, and GitHub Actions on a daily basis. Assigning
rustatian
as both reviewer and assignee ensures that updates are promptly reviewed. However, consider adding more team members as reviewers and assignees to distribute the workload and reduce bottlenecks.tests/mock/logger.go (1)
- 17-23: The
ZapTestLogger
function correctly sets up a mock logger for testing purposes. It's a good practice to isolate logging in tests to prevent cluttering the output and to focus on test-related logs only. This implementation aligns well with the testing strategy.plugin.go (2)
- 39-39: The check for the existence of the plugin configuration using both
pluginName
andmasterPluginName
is a good practice to ensure that the plugin is only initialized when its configuration is present. This prevents unnecessary initialization and potential errors in environments where the plugin is not used.- 61-65: The refactoring of function signatures to use
jobs.Queue
instead ofpq.Queue
and the adjustments inDriverFromConfig
andDriverFromPipeline
functions are in line with the PR's objectives to integrate with the Roadrunner server's job processing capabilities. Ensure that all references to these functions are updated accordingly in the codebase..github/workflows/codeql-analysis.yml (1)
- 1-73: The CodeQL workflow is correctly set up to perform code analysis on push and pull request events targeting the
master
branch, as well as on a weekly schedule. The configuration specifies Go as the language for analysis, which aligns with the project's technology stack. Ensure that the project's build process is compatible with the Autobuild step or provide custom build commands if necessary.pubsubjobs/item.go (6)
- 4-6: Imports
strconv
,sync
, andsync/atomic
have been added. Ensure these packages are necessary for the functionality being implemented, particularly if they introduce concurrency or atomic operations which can complicate the codebase.- 26-26: The struct field
Headers
has been renamed toMetadata
. This change should be clearly documented in the PR to avoid confusion and ensure that all references to this field throughout the codebase are updated accordingly.- 64-66: The addition of the
GroupID
method. Confirm that this method's implementation aligns with its intended use case and that thePipeline
field is the correct source for a group ID.- 73-74: The
Headers
method has been renamed to align with theMetadata
field change. Ensure that this renaming is consistent across the codebase and that all references to the old method name are updated.- 150-158: The
fromJob
function converts ajobs.Message
to anItem
. Ensure that this conversion correctly handles all fields, especially the newMetadata
field, and that it properly initializes theOptions
struct with the correct values from the job message.- 244-250: The
stob
(string to bool) function has been added. Verify that this function correctly handles the conversion of string values to boolean, especially considering case sensitivity and potential string values that should be interpreted astrue
orfalse
.tests/mock/observer.go (6)
- 1-22: The file header indicates that this is a mock implementation for logging purposes, derived from Uber Technologies, Inc. Ensure that the license and copyright notice are appropriate and that the use of this mock adheres to the terms specified.
- 31-45: The
ContextMap
method in theLoggedEntry
struct converts the context fields into a map. Verify that this conversion correctly handles all field types and that the map representation is used appropriately in tests or logging.- 91-110: Filter methods (
FilterLevelExact
,FilterMessage
,FilterMessageSnippet
, etc.) have been added to theObservedLogs
struct. Ensure that these methods correctly filter the logs based on the specified criteria and that they are used appropriately in tests to verify logging behavior.- 136-149: The
Filter
method provides a generic way to filter observed logs based on a provided function. Review this method for correctness and ensure that it is used appropriately in tests to filter logs based on complex criteria.- 157-165: The
New
function creates a new mock logging core and associated observed logs. Ensure that this function is correctly initializing the mock logger and that it is used appropriately in tests to capture and analyze logged entries.- 188-195: The
Write
method in thecontextObserver
struct is responsible for adding logged entries to the observed logs. Review this method for correctness, especially in terms of how it combines context fields with log fields, and ensure that logged entries are correctly captured in tests.tests/helpers/helpers.go (7)
- 29-44: The
ResumePipes
function is a test helper for resuming specified pipelines. Ensure that this function correctly interacts with the job system via RPC calls and that error handling is appropriately implemented.- 47-58: The
PushToPipe
function is a test helper for pushing jobs to a specified pipeline. Review the job creation process within this function to ensure that it correctly sets up jobs for testing, including theautoAck
flag and other job options.- 61-82: The
PushToPipeDelayed
function allows for pushing jobs with a specified delay. Verify that this function correctly calculates and applies the delay to jobs and that it interacts properly with the job system via RPC calls.- 100-115: The
PausePipelines
function is a test helper for pausing specified pipelines. Ensure that this function correctly interacts with the job system to pause pipelines and that error handling is appropriately implemented.- 118-140: The
DestroyPipelines
function is a test helper for destroying specified pipelines. Review the retry logic within this function to ensure that it correctly handles temporary failures and that pipelines are properly destroyed in the job system.- 143-164: The
Stats
function retrieves the state of the job system for specified pipelines. Ensure that this function correctly gathers and returns the expected state information and that it is used appropriately in tests to verify the state of the job system.- 167-187: The
DeclarePipe
function is a test helper for declaring a new pipeline with specified options. Review this function for correctness in setting up pipelines, especially the configuration options, and ensure that it interacts properly with the job system.pubsubjobs/driver.go (3)
- 5-6: Imports
strconv
andstrings
have been added. Ensure these packages are necessary for the functionality being implemented, particularly if they introduce operations on strings or conversions that could affect performance or correctness.- 97-135: > 📝 NOTE
This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [64-125]
The
FromConfig
function initializes the driver from configuration. Review this function for correctness in handling configuration values, especially error handling and the initialization of the Google Pub/Sub client. Ensure that the configuration is correctly applied and that any errors are appropriately handled.
- 152-384: > 📝 NOTE
This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [131-181]
Similar to
FromConfig
, theFromPipeline
function initializes the driver from a pipeline configuration. Review this function for correctness, especially in terms of error handling and the application of pipeline-specific configurations.tests/jobs_test.go (3)
- 299-299: The use of a mock logger (
mocklogger.ZapTestLogger
) is a good practice for capturing and asserting log output in tests. However, ensure that the logger is adequately capturing all relevant log levels and contexts needed for the assertions later in the test.- 366-366: The assertion
require.Equal(t, 4, oLogger.FilterMessageSnippet("auto ack is turned on, message acknowledged").Len())
checks for a specific number of log messages. This can make the test brittle if the implementation changes slightly but still behaves correctly. Consider if there's a more robust way to assert the desired behavior without relying on the exact number of log messages.- 454-459: The assertions in
TestRemovePQ
rely heavily on specific log messages. This ties the test's correctness to the logging implementation, which might change over time. If possible, use more direct ways of verifying the system's state or behavior, such as querying the state of the pipeline or the jobs directly.
Reason for This PR
closes: roadrunner-server/roadrunner#1471
Description of Changes
MVP for google pub sub plugin
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the MIT license.
PR Checklist
[Author TODO: Meet these criteria.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]
git commit -s
).CHANGELOG.md
.Summary by CodeRabbit
New Features
.gitignore
to better manage project files.Chores
Documentation